Self-control in Sparsely Coded Networks

A complete self-control mechanism is proposed in the dynamics of neural networks through the introduction of a time-dependent threshold, determined in function of both the noise and the pattern activity in the network. Especially for sparsely coded models this mechanism is shown to considerably improve the storage capacity, the basins of attraction and the mutual information content of the network. Sparsely coded models have attracted a lot of attention in the development of neural networks, both from the device oriented and biologically oriented point of view [1]– [2]. It is well-known that they have a large storage capacity, which behaves as 1/(a ln a) for a small where a is the pattern activity. However, it is clear that the basins of attraction, e.g., should not become too small because then sparse coding is, in fact, useless. In this context the necessity of an activity control system has been emphasized, which tries to keep the activity of the network in the retrieval process the same as the one for the memorized patterns [3]– [5]. This has led to several discussions imposing external constraints on the dynamics (see the references in [2]). Clearly, the enforcement of such a constraint at every time step destroys part of the autonomous functioning of the network. An important question is then whether the capacity of storage and retrieval with non-negligible basins of attraction can be improved and even be optimized without imposing these external constraints, keeping at the same time the simplicity of the architecture of the network. In this Letter we answer this question by proposing , as far as we are aware for the first time, a complete self-control mechanism in the dynamics of neural networks. This is done through the introduction of a time-dependent threshold in the transfer function. This threshold is chosen as a function of the noise in the system and the pattern activity, and adapts itself in the course of the time evolution. The difference with existing results in the literature [2] precisely lies in this adaptiv-ity property. This immediately solves, e.g., the difficult problem of finding the mostly narrow interval for an optimal threshold such that the basins of attraction of the memorized patterns do not shrink to zero. We have worked out the practical case of sparsely coded models. We find that the storage capacity, the basins of attraction as well as the mutual information content are improved. …